Design and Evaluation Challenges of Conversational Agents in Health Care and Well-being: Selective Review Study.

BACKGROUND: Health care and well-being are 2 main interconnected application areas of conversational agents (CAs). There is a significant increase in research, development, and commercial implementations in this area. In parallel to the increasing interest, new challenges in designing and evaluating CAs have emerged. OBJECTIVE: This study aims to identify key design, development, and evaluation challenges of CAs in health care and well-being research. The focus is on the very recent projects with their emerging challenges. METHODS: A review study was conducted with 17 invited studies, most of which were presented at the ACM (Association for Computing Machinery) CHI 2020 conference workshop on CAs for health and well-being. Eligibility criteria required the studies to involve a CA applied to a health or well-being project (ongoing or recently finished). The participating studies were asked to report on their projects' design and evaluation challenges. We used thematic analysis to review the studies. RESULTS: The findings include a range of topics from primary care to caring for older adults to health coaching. We identified 4 major themes: (1) Domain Information and Integration, (2) User-System Interaction and Partnership, (3) Evaluation, and (4) Conversational Competence. CONCLUSIONS: CAs proved their worth during the pandemic as health screening tools, and are expected to stay to further support various health care domains, especially personal health care. Growth in investment in CAs also shows the value as a personal assistant. Our study shows that while some challenges are shared with other CA application areas, safety and privacy remain the major challenges in the health care and well-being domains. An increased level of collaboration across different institutions and entities may be a promising direction to address some of the major challenges that otherwise would be too complex to be addressed by the projects with their limited scope and budget.

A review of mobile apps for epilepsy self-management.

Mobile health app developers increasingly are interested in supporting the daily self-care of people with chronic conditions. The purpose of this study was to review mobile applications (apps) to promote epilepsy self-management. It investigates the following: 1) the available mobile apps for epilepsy, 2) how these apps support patient education and self-management (SM), and 3) their usefulness in supporting management of epilepsy. We conducted the review in Fall 2017 and assessed apps on the Apple App Store that related to the terms "epilepsy" and "seizure". Inclusion criteria included apps (adult and pediatric) that, as follows, were: 1) developed for patients or the community; 2) made available in English, and 3) less than $5.00. Exclusion criteria included apps that were designed for dissemination of publications, focused on healthcare providers, or were available in other languages. The search resulted in 149 apps, of which 20 met the selection criteria. A team reviewed each app in terms of three sets of criteria: 1) epilepsy-specific descriptions and SM categories employed by the apps and 2) Mobile App Rating Scale (MARS) subdomain scores for reviewing engagement, functionality, esthetics, and information; and 3) behavioral change techniques. Most apps were for adults and free. Common SM domains for the apps were treatment, seizure tracking, response, and safety. A number of epilepsy apps existed, but many offered similar functionalities and incorporated few SM domains. The findings underline the need for mobile apps to cover broader domains of SM and behavioral change techniques and to be evaluated for outcomes.

Multicenter clinical assessment of improved wearable multimodal convulsive seizure detectors.

OBJECTIVE: New devices are needed for monitoring seizures, especially those associated with sudden unexpected death in epilepsy (SUDEP). They must be unobtrusive and automated, and provide false alarm rates (FARs) bearable in everyday life. This study quantifies the performance of new multimodal wrist-worn convulsive seizure detectors. METHODS: Hand-annotated video-electroencephalographic seizure events were collected from 69 patients at six clinical sites. Three different wristbands were used to record electrodermal activity (EDA) and accelerometer (ACM) signals, obtaining 5,928 h of data, including 55 convulsive epileptic seizures (six focal tonic-clonic seizures and 49 focal to bilateral tonic-clonic seizures) from 22 patients. Recordings were analyzed offline to train and test two new machine learning classifiers and a published classifier based on EDA and ACM. Moreover, wristband data were analyzed to estimate seizure-motion duration and autonomic responses. RESULTS: The two novel classifiers consistently outperformed the previous detector. The most efficient (Classifier III) yielded sensitivity of 94.55%, and an FAR of 0.2 events/day. No nocturnal seizures were missed. Most patients had <1 false alarm every 4 days, with an FAR below their seizure frequency. When increasing the sensitivity to 100% (no missed seizures), the FAR is up to 13 times lower than with the previous detector. Furthermore, all detections occurred before the seizure ended, providing reasonable latency (median = 29.3 s, range = 14.8-151 s). Automatically estimated seizure durations were correlated with true durations, enabling reliable annotations. Finally, EDA measurements confirmed the presence of postictal autonomic dysfunction, exhibiting a significant rise in 73% of the convulsive seizures. SIGNIFICANCE: The proposed multimodal wrist-worn convulsive seizure detectors provide seizure counts that are more accurate than previous automated detectors and typical patient self-reports, while maintaining a tolerable FAR for ambulatory monitoring. Furthermore, the multimodal system provides an objective description of motor behavior and autonomic dysfunction, aimed at enriching seizure characterization, with potential utility for SUDEP warning.

Personal discovery in diabetes self-management: Discovering cause and effect using self-monitoring data.

OBJECTIVE: To outline new design directions for informatics solutions that facilitate personal discovery with self-monitoring data. We investigate this question in the context of chronic disease self-management with the focus on type 2 diabetes. MATERIALS AND METHODS: We conducted an observational qualitative study of discovery with personal data among adults attending a diabetes self-management education (DSME) program that utilized a discovery-based curriculum. The study included observations of class sessions, and interviews and focus groups with the educator and attendees of the program (n = 14). RESULTS: The main discovery in diabetes self-management evolved around discovering patterns of association between characteristics of individuals' activities and changes in their blood glucose levels that the participants referred to as "cause and effect". This discovery empowered individuals to actively engage in self-management and provided a desired flexibility in selection of personalized self-management strategies. We show that discovery of cause and effect involves four essential phases: (1) feature selection, (2) hypothesis generation, (3) feature evaluation, and (4) goal specification. Further, we identify opportunities to support discovery at each stage with informatics and data visualization solutions by providing assistance with: (1) active manipulation of collected data (e.g., grouping, filtering and side-by-side inspection), (2) hypotheses formulation (e.g., using natural language statements or constructing visual queries), (3) inference evaluation (e.g., through aggregation and visual comparison, and statistical analysis of associations), and (4) translation of discoveries into actionable goals (e.g., tailored selection from computable knowledge sources of effective diabetes self-management behaviors). DISCUSSION: The study suggests that discovery of cause and effect in diabetes can be a powerful approach to helping individuals to improve their self-management strategies, and that self-monitoring data can serve as a driving engine for personal discovery that may lead to sustainable behavior changes. CONCLUSIONS: Enabling personal discovery is a promising new approach to enhancing chronic disease self-management with informatics interventions.

Structured scaffolding for reflection and problem solving in diabetes self-management: qualitative study of mobile diabetes detective.

OBJECTIVE: To investigate subjective experiences and patterns of engagement with a novel electronic tool for facilitating reflection and problem solving for individuals with type 2 diabetes, Mobile Diabetes Detective (MoDD). METHODS: In this qualitative study, researchers conducted semi-structured interviews with individuals from economically disadvantaged communities and ethnic minorities who are participating in a randomized controlled trial of MoDD. The transcripts of the interviews were analyzed using inductive thematic analysis; usage logs were analyzed to determine how actively the study participants used MoDD. RESULTS: Fifteen participants in the MoDD randomized controlled trial were recruited for the qualitative interviews. Usage log analysis showed that, on average, during the 4 weeks of the study, the study participants logged into MoDD twice per week, reported 120 blood glucose readings, and set two behavioral goals. The qualitative interviews suggested that individuals used MoDD to follow the steps of the problem-solving process, from identifying problematic blood glucose patterns, to exploring behavioral triggers contributing to these patterns, to selecting alternative behaviors, to implementing these behaviors while monitoring for improvements in glycemic control. DISCUSSION: This qualitative study suggested that informatics interventions for reflection and problem solving can provide structured scaffolding for facilitating these processes by guiding users through the different steps of the problem-solving process and by providing them with context-sensitive evidence and practice-based knowledge related to diabetes self-management on each of those steps. CONCLUSION: This qualitative study suggested that MoDD was perceived as a useful tool in engaging individuals in self-monitoring, reflection, and problem solving.

Participatory approach to the development of a knowledge base for problem-solving in diabetes self-management.

OBJECTIVE: To develop an expandable knowledge base of reusable knowledge related to self-management of diabetes that can be used as a foundation for patient-centric decision support tools. MATERIALS AND METHODS: The structure and components of the knowledge base were created in participatory design with academic diabetes educators using knowledge acquisition methods. The knowledge base was validated using scenario-based approach with practicing diabetes educators and individuals with diabetes recruited from Community Health Centers (CHCs) serving economically disadvantaged communities and ethnic minorities in New York. RESULTS: The knowledge base includes eight glycemic control problems, over 150 behaviors known to contribute to these problems coupled with contextual explanations, and over 200 specific action-oriented self-management goals for correcting problematic behaviors, with corresponding motivational messages. The validation of the knowledge base suggested high level of completeness and accuracy, and identified improvements in cultural appropriateness. These were addressed in new iterations of the knowledge base. DISCUSSION: The resulting knowledge base is theoretically grounded, incorporates practical and evidence-based knowledge used by diabetes educators in practice settings, and allows for personally meaningful choices by individuals with diabetes. Participatory design approach helped researchers to capture implicit knowledge of practicing diabetes educators and make it explicit and reusable. CONCLUSION: The knowledge base proposed here is an important step towards development of new generation patient-centric decision support tools for facilitating chronic disease self-management. While this knowledge base specifically targets diabetes, its overall structure and composition can be generalized to other chronic conditions.